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A Methodological Mote on Contextual Effects Studies in Education 

Abstract 

This paper suggests four sources of inconclusive results in studies of 
school effects: Poor definitions of school context, poor sampling techniques, 

lack of attention to techniques of partitioning variance, and lack of awareness 
of the effects of changing units of analysis on the size of statistical associa 
tions. Each problem is briefly described, and strategies for surmounting it 
are outlined. 
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A Methodological Note on Contextual Effects Studies in Education 

For some time, the impact of schools on students has been the subject of 
intensive investigation by sociologists, and more recently, by economists. To 
date, the results of this research have been ambiguous. A quick overview of 
several studies (Boyle, 1966a; Coleman, 1961; Coleman et al. 1966; Rrauss, 1964, 
Turner, 1964; Wilson, 1959, Bowles and Levin, 1968a, 1968b, Cambell and Alexander, 
1965; Duncan, Haller and Portes, 1968; Hauser, 1969, 1970a, 1970b; McDill, Rigsby, 
Meyers, 1969; McDill, Meyers, Rigsby, 1967; Swell and Arraer, 1966a; Mood, 1970; 
Riesling, 1967; Nichols, 1964; Astin, 1963; Meyer, 1970; Boyle, 1966a) indicates 
that there are fairly wide ranges of opinion about the importance of school 
effects on students. In some cases, there has been a fairly extensive debate 
over the size and importance of school effects (see, for example, Bowles and 
Levin 1968a, 1966b) vs. Coleman, (1968), and Smith (1968). This debate was also 
joined by Cain and Watts (1968). One might also look at Hauser (1970a, 1970b) 

, vs . Barton (1970), or Swell and Armer (1966a, 1966b) vs. Turner (1966), Michael 
(1966), and Boyle (1966b). 

The very fact that a controversy exists is a striking finding. In an 
enterprise as expensive as the public school system, with educational inequalities 
of a fairly extensive nature, it ought to be possible to show consistent returns 
to different types of educational practices. The purpose of this paper is to 
discuss the inability of sociologists to show such returns, and to suggest some 
research strategies which might help to clarify the problem. The paper deals 
with four problems which are important in the study of school effects: the 

definition of school context, the sample of schools selected for a study, 
partitioning of variance in outcome measures, and the unit of analysis selected 
for study. 
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I. Definition of the Context 

The contextual hypothesis examined by sociologists is that there are differ- 
ences between schools which have an impact on educational outcomes for individuals 
The impact of these differences will be over and above any impact attributable 
to individual characteristics of the pupils. 

An example of this model (following Rogoff, 1961), is as follows: 

(a) Ecological processes result in the segregation of socioeconomic groups 
within a community. 

(b) The resulting segregation has the effect of segregating schools along 
socioeconomic lines. 

(c) Attitudes and values, and specifically educational attitudes and values, 
vary among social classes. 

(d) Thus, schools and neighborhoods will vary in modal aspirational levels 
and normative social climates. 

(e) Schools (and school attendance areas) represent functioning sub- 

communities for students, and the characteristic attitudes and values 
of communities Influence their members. 

(f) Thus, modal aspirational levels and normative social climates of 
schools and their attendance areas affect all pupils, over and above 
the effects of students' own abilities, socioeconomic levels, aspira- 
tions, etc. For example, the aspirations of lower-class children in a 
predominantly middle class school will be higher than those of similar 
oliildren attending predominantly lower-class schools. 

There is a major flaw in describing school characteristics as aggregates of 
individual characteristics, especially as aggregates of individual social class. 
Aggregates such as socio-economic -status, are proxy variables standing for events 
which are supposed to impinge on the lives of people (do the parents read to 
their children?). In order to understand what brings about an association (for 
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e xamp le) between socio-economic status and some output such as reading ability, 
one must resort to the variables for which SES is a proxy. This can be done 
inferentially with some degree of confidence for individuals, but the degree of 
inference required to make generalizations about school averages, is large. 

Perhaps because of this many studies have found small associations between aver- 
age contexts and some form of individual outcome. 

An alternative contextual hypothesis commonly found in studies which attempt 
to discover effective resource allocation, is that better teachers produce better 
students. Since better teachers are paid more than poor teachers, there ought to 
be a positive association between teacher salary and pupil achievement over and 
above influences of individual characteristics of the pupils. 

Again, ‘a tremendous degree of inference is operating in the model. Teachers 
are usually paid according to their level education and their experience. There 
is no reason to assume that either of these variables ±& associated with good 
teaching. 

If contextual effects studies are to produce useful results, a more detailed 
analysis of the educational inputs is needed. The futility of present studies 
is made apparent by imagining that a significant and indisputably large associa- 
tion was found between (for example) the average social class of a schools' 
students and pupil achievement. Aside from labelling the situation, what does 
this finding tell the school administrator. Given the political realities of 
public education- -which make it difficult fc* c«ieu advantage of findings showing 
that it i r ■ »ndonireahl« to segregate students along racial or class lines-- 
administrators need more detailed information about the causal mechanisms under- 
lying these finding so that they can formulate appropriate policies for their 
own schools. 

How might one go about providing more useful data to both the practitioner 
and the theorist? One way, would be to examine variables which differ from 
school to school, which are organizational properties of the school, and which 
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have some theoretical relevance to the outcome in which the researcher is inter- 
ested. 

For example, if one were to find the contextual relationship outlined above, 
would it not seem reasonable to look for organizational characteristics which 
differed from low SES to high SES schools? These might provide some useful in- 
sight into the causes of the observed phenomenom. If teacher salaries were 
associated with pupil achievement, would it not be reasonable to try and discover 
what high paid teachers did that low paid teachers did not do? Perhaps then it 
would be possible to provide data of theoretical and practical significance. 

In sum, definitions of school contexts bases on macro or aggregate data will 
probably not provide useful information on the causal mechanisms by which school 
effects act. Variables which are more closely tied to the operation of schools 
are necessary. 

II. The Cross-Sectional Sample of Schools 

Even with an adequate definition of school context, it is difficult to be 
certain that pupils in a study have been under the influence of a given school 
effect for a meaningful length of time. Mobility of pupils from one school to 
another is a large contributor to this problem but an even more serious source 
is that students may move from one organizational context to another as they 
change grades (or for that matter, classes). 

Further, the impact of school structure is apt to be cummulative, and to be 
more important during some parts of the school career than others (see Wilson, 
1967: 171-174, 179, 190, for example). For this reason, cross-sectional studies 

should probably be used only to generate useful conceptions of school structure. 
Longitudinal studies with continual assessment of the sample schools's organiza- 
tional structure, could then be used to generate the data for a good test of a 
contextual effects hypothesis. 
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III. Partitioning of Variance 

The major analytical tool used by students of contextual effects has been 
ordinary least-squares regression analysis. The model is usually of the general 
form: 

OUT » f (CON + IBV ) (2.1) 

ij j i 

Where OUT is an educational outcome for individual i in school j 
CON is a measure of context for school j 

IBV is a set of background variables (such as sex, IQ) for individual i. 
Analysts search for the association between educational outcomes and school 
contexts, holding constant the effects of background variables. In practice, this 
involves a comparison between (2.1) and a second regression model: 

CON - f (IBV ) (2.2) 

ij i 

Differences in the predictive efficiency of (2.1) and (2.2) are to 

school contexts. Frequently these aw omall. and researchers often conclude that 

school effects are insignificant. 

One could be more confident of this result were researchers to (a) apportion 
variance between the School Context and BackgroundVariables and (b) report the 
maximum amount of variance in the Outcome which could be attributed to _any_ dif- 
ferences between schools in the first place. These differences, it should be 
noted, not only include such "sociological" things as socio-economic and racial 
composition of the school, they also include "educational' things such as 
curriculum and teaching strategies. 

Taking the, second point first, the appropriate regression model is of the 
form: 

OUT = f(S + S .... + S ) (2.3) 

ij 1 2 j 

Where OUT is an educational outcome for individual i in school j 
S is one if person i is in school j, zero otherwise 

j 
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In short, the model is an attempt to predict an outcome solely from knowledge of 
the school in which the student is located.^ 

This model is of crucial importance, since it yields the upper limit of the 
amount of variance in the outcome which can be explained by any sort of variance 
in school contexts in the sample of schools. 

Having computed equation (2.3), a researcher knows how much variance in an 
outcome could be attributed to differences between schools. Assuming that this 
is large enough to be of interest, he then wishes to know how much of that variance 
can be attributed to his measure of the school context. 

This estimate is obtained by replacing the l's in model (2.3) with estimates 
of the school context. Then, by collecting terms one arrives at a regression 
model: 

OUT = f (CON ) (2.4) 

ij j 

Where OUT is an educational outcome for individual i in school j 

ij 

CON the measure of school context for school j 

j 

This model yields the amount of variance in the outcome which can be accounted 
for by differences in school context, neglecting the possible influence of 
controls. 

The advantage of these procedures is readily shown by an example. Using data 
from a study of Bureaucracy and Alienation, measures of Alienation from School 
(OUT), School Bureaucratization (CON), and students' socio-economic background 
(IB\) were computed. 

lxhis regression model uses a series of "dummy variables" to predict the outcome. 
Any other analysis of variance technique which revealed the amount of variance 
between schools would be appropriate. 



Means, standard deviations and intercorrelations were then calculated for 
these variables (and for others to be used later in this paper). * The results 
are shown in Table 1. 

INSERT TABLE I HERE 

The four regression models outlined above were then computed from the data 
in Table I. The results of this analysis are shown in Table II. 

INSERT TABLE II HERE 

Data continued in Table II heads off a number of issues which are often de- 
bated in the contextual effects literature. First, an examination of column 2.3 
shows that only 6.4% of the variance in Alienation from School is attributable to 
differences between schools. Since this is the upper limit on the amount of 
variance which can be explained by differences between schools, we are spared a 

lThe data were collected from a random sample of schools located in Ontario, 

Canada. The sample was stratified on the basis of school district size, school 
type and location (rural or urban). One thousand, eight hundren and ninety grade 
10 students provided acceptable data dealing with their perceptions of school 
bureaucratization, feelings of alienation from school, and their parents' 
occupations. A second set of 1,890 students were used as a basis for computing 
aggregate Bureaucracy and Alienation scores for each school. The bureaucracy 
scale reported here measures attempts by school authorities to control the 
behavior of students and is derived by extensive modification of Hall's (1961) 
measure of bureaucratization. The alienation measure is a composite of Seeman's 
(1959) five dimensions of alienation. The measure of social class is an adaptation 
of the Blishen Scale, (Blishen, 1965) a socio-economic status index developed 
for use in Canada. More complete descriptions of the development of the aliena- 
tion and bureaucracy scales may be found in Anderson (1970). 



debate over the causes of a small amount of variance being explained by the 
predictors. The explanation is simply that there are few dif ferep.ces in aliena- 
tion attributable to differences between the schools in the ecunple. 

The state of affairs revealed by 2.3 is unde%ireable from the viewpoint of 
the contextual effects researcher, but it may well be quite common in education. 

It is most probable that there is much more variation between children than 
between schools on most educational outputs, so one would expect a relatively 
small amount of the total variance to be explained by differences between schools. 

Model 2.4 (Table II) shows the effect of restricting differences between 
schools to differences in school bureaucratization. The difference between this 
model and model 2.3 reflects the fact that restricting differences between schools 
to differences in Bureaucrati zation reduces the variance of school context. In 
other words, it shows the extent to which this one contextual variable under- 
specifies the school context. Contextual effects researchers are prone to stumble 
on this point, since they seem oblivious to the relationship between 2.4 and 2.3. 

Models 2.1, 2.2, and 2.4 can be used to obtain estimates of the independent 
effects of context and background variables on the outcome (Mood, 1971). From 
2.1 we see that school bureaucratization (CON) and socio-economic status- (IBV) 
account for 4.3% of the variance in alienation (OUT). The independent effect of 
bureaucratization is found by comparing 2.1 and 2.2, the difference between the 
two models (3.3%) is the independent effect of bureaucratization. The independent 
effect of socio-economic status (negligible) is found by comparing 2.1 and 2.4. 
Overlap between bureaucratization and social class (owing to the fact that the 
two are correlated) is found by adding 2.2 to 2.4 and subtracting the resulting 
total from 2.1. In this case, 1% of the variance in alienation from school is 
attributable to overlap (or multi-colinearity) between the two independent 



variables. 



TABLE I 

MEANS, STANDARD DEVIATIONS AND CORRELATIONS BETWEEN ALL VARIABLES 
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Standard Deviations .999 .253 1.915 



TABLE II 



REGRESSION EQUATIONS PREDICTING INDIVIDUAL ALIENATION 
FROM SCHOOL CONTEXT AND SOCIAL CLASS 



Equations 

Predictors 

20 Li 2_J3 2A 



CON 




.fill 


IBV 




-.012 


School 


1 





2 

3 

4 

5 

6 

7 

8 
9 

10 

11 

12 

13 

14 

15 

16 
17 
18* 

RSQ -043 



.710 



-.053 

-.795 

-.579 

-.260 

-.495 

-.305 

-.195 

-.628 

.015 

.159 

.050 

-.417 

.000 

-.224 

-.249 

.000 

-.464 

.134 

.000 

.010 .064 .043 



Independent CON = ((2.1) - (2.2)) =3.37. (p. - -OOM) 
Independent IBV = ((2.1) - (2.4)) = 0.07. (p. = .3176) 



Overlap = ((2.4 +2.2) - (2.1)) " 1% 



★All 18 schools were included. This is possible because 
iterative regression techniques, described by Greenbcrger 
and Ward (1956) and Ward (1962). were used. 
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Careful partitioning of variance, as outlined above, does much to aid under- 
standing of contextual effects, especially in cases where overlap is high. 
Ultimately, there is little that can be done about the problem of multi- 
colinearity other than to seek situations in which it is possible to maximize 
variance in the school characteristic being studied while randomizing out some 
of the most troublesome interferring variables. An example might be drawn from 
studies which show a high overlap between the average salary of teachers and the 
average SES of students. If one wished to study the effect of teacher salary 
independent of student SES, one might seek situations where teacher salary varied 
independent of social class. High salaries might be found in both high SES 
neighborhoods and areas where there was a large amount of federal aid. Low 
salaries might be sought in wealthy areas as well as in poor neighborhoods. 

Such a sample might minimize the overlap between SES and teacher salary, and 
allow a researcher to study the independent effect of eac. more effectively than 
could be done in a random sample. 

Before leaving the issue of partitioning variance, a word about control 
variables is in order. The typical contextual effects study controls for the 
quality of a student input to the system, and seeks school effects over and above 

these variables. 

Since all students do not have the same characteristics, a more appropriate 
analysis procedure calls for separate analyses for different types of student 
wherever "type of student" is thought to be an important variable. Coleman for 
example, finds quite different regression equations for different ethnic groups 
when he attempts to predict verbal achievement from school and background 
variables (1966: 299-306). This fact could have been lost had Coleman merely 



controlled for ethnic origin.. 
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IV. The Unit of Analysis 

Tables I and II contain two units of analysis: individuals and groups. It 

has been known for some time that changing units of analysis alters the size of 
correlation coefficients and a good deal of work has been done in organizing 
the theoretical and methodological implications of this fact. 1 However, little 
seems to have been done to examine the effect of mixing units of analysis in the 
same study, yet this is by far the most common form of analysis used by students 
of contextual effects. 

Basically, one finds two units of analysis in sociological research: the 

individual and some form of group. This leads to four pair-wise combinations of 

2 

independent and dependent variables: 



Dependent 

(1) Individual Outcomes 

(2) Group Outcomes 

(3) Individual Outcomes 

(4) Group Outcomes 



Independent 
Individual Attributes 
Individual Attributes 
Group Attributes 
Group Attributes 



Contextual effects studies are usually a combination of (1) and (3), rhnt is, 
they use individual and group attributes to examine individual outcomes. To 
systematically explore grouping in these studies, fnm* models may be investigated 



with the 


data described above: 


3.1) 


OUT 


= f (IPCON + IBV ) 




ij 


ij ij 


3.2) 


OUT 


= f (CON + IBV ) 




ij 


j ij 


3.3) 


OUT 


= f (CON + GBV ) 




ij 


j j 


Hannan 


(1970) 


offers an excellent overview 



^Riley (1964) expanda this issue. 
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3.4) GOUT .« f (CON + GBV ) 

j j J 

Where OUT = an outcome for person i in school j 

GOUT = average outcome for all students in school j 
IPCON = individual i’s perception of the context of school j 
CON = a measure of the context of school j 
IBV = a background variable for person i in school j 
GBV * average background measure for all studeuts in school j 
Model 3.1 is rarely seen in contextual effects studies because the context 
is not normally assumed to have an "existence" at the individual level. None- 
theless, its inclusion proves illuminating in the study of aggregation. Model 3.2 
is what might be called the traditional contextual effects model, while model 3.3 
assumes two different measures of context, an organizational property and an 
aggregation of individual properties. Model 3.4 used if schools are the unit of 
analysis in a study. 

The application of these models to the data are shown in Table 111. As can 
be seen from the Table, there are very marked^changss in the amount of variance 

y 

INSERT TABLE III HERE 

explained by school context and social class as the units of analysis change. At 
individual levels (model 3.1a) school context explains 27.3% of the between - 
student variance in alienation. However, aggregating the school effect alone 
(model 3.2a) reduces this to only 4. 37.. 

Aggregating both the school context and the background variable by school 
v'uh-joI J.3 increases the amount of between-student variance explained by the 
full model to 4.97., but reduces the independent contribution of the original 
measure of school context. However in this model the aggregated background 
variables in another measure of context, and school differences of any set can 
explain no more than 6.47. of the variance in individual alienation. Aggregating 



REGRESSION EQUATIONS PREDICTING INDIVIDUAL AND GROUP ALIENATION 
FROM SCHOOL CONTEXT AND SOCIAL CLASS 
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All RSQ's are significant (p. £ .01). 
Significant Beta weight (p. £ .01). 
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both the dependent end independent variables (model 3.4a) reises the verience 
explained to a full 75.7%, but now the model explains between school variance. 
Thus, model 3.4 explains 75.7% of the 6.4% explained by model 2.3. 

Which of these models are we to use in studying school effects? It could 
be argued that one should use the aggregate model, since it produces the "largest" 
effect. But at tha very least, Robinson's (1950) paper should alert us to the 
possibility that such a large squared multiple correlation may be an overstatement 
of the association between school properties and ind?.vidual results. On the other 
hand, use of the individual - individual model is complicated by three problems: 
the probability of response bias when an individual provides perceptual data on 
both sides of the regression equation, the likelihood that data obtained on the 
individual level are either not suitable or are unavailable for the analysis 
and, most important, the fact that individual - individual models do not reflect 
the theoretical model which deals with organizational impacts on individual out- 
comes. The latter point is important, for not only do the sizes of correlations 
change, in some studies their sign has changed when the units of analysis are 

altered.* 

Looking over Table III one may note that, while the squared multiple correla- 
tion is undergoing very substantial changes, the unstandardized regression 
weights attached to each variable remain comparatively stable. Here we may see 
a key to the interpretation of contextual effects data. 

The regression weight is the amount of change in the dependent varibble 
which may be expected to arise per unit change in the independent variable. 
Weights in equations involving more than one variable show the amount of change 
to be expected in the outcome per unit change in the independent variable in the 
presence of other factors in the equation. Considering the aggregation which has 




^ee Bon jean and Grimes, 1970: 368, for an example. 
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been performed in moving from 3.1 to 3.3 or 3.4, the regression weights have 
remained quite stable. They are simple to interpret, and have some practical 
meaning . 

The regression equation, and not the amount of variance explained, seems to 
be the appropriate means of seeking the size of school effects. In fact, this 
is precisely what Blalock means when he notes that 

...if there is any degree of assurance that in effect, manipulations 
have been made primarily in terms of the independent variable;, then 
comparisons involving slopes will ordinarily be more meaningful than 
those using correlation coefficients, .. .Basically, our interest in 
correlation coefficients in these comparisons should be mainly to 
help us determine the degree of accuracy in our estimates of slopes. 

(Blalock, 1964: 126, emphasis added). 

Given this interpretation, then, one would assume that the school effects found 
in these data were significant, but that they could not be expected to be the 
same for all individuals in each school. Such an interpretation seems much more 
reasonable than the dismissal of school effects entirely on the grounds that they 
explain a trivial amount of the variance in alienation. In fact, the amount of 
variance attributable to school effects is very dependent upon the units of 
analysis selected for a study, while the regression weights are apparently much 

t t 

more stable across levels of aggregation. 



Implications 

Three factors have been suggested for the results n £ contextual 

offecLS sLudies: rhpir r -a theoretically and practically useless variables, 

their failure to study and partition variance, reliance upon squared multiple 
correlations fimii reero.c«J«>n equations as measures of the importance of school 



effects. 
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To overcome these problems, three or four research strategies may be proposed. 
First, researchers ought to formulate their studies in terms of variables which 
have relevance in a causal sense (i.e., which are more than descriptive of the 
statu s quo) . Second, samples ought to be chosen so as to maximize variance in 
the school context under study, while randomizing or better yet blocking on, 
variables which would tend to be correlated with the school context in a random 
sample. Further, longitudinal, as opposed to cross-sectional , studies are re- 
quired. Third, the extent to which data allow statements about the independent 
effects of variables ought to be examined. This requires an attempt to determine 
the independent contribution of each variable to the variance explained. The more 
overlap between variables, the less able one is to specify the correct regression 
weights for any variable in the regression equation. The type of partitioning 
recommended by Mood (1971) is probably appropriate for this sort of analysis. 
Fourth, researchers ought to reduce their emphasis on the squared multiple 
correlation coefficient as the role estimate of the strength of the relationship 
between a criterion and a set of predictors. A better estimate of the importance 
of an independent variable is found by regression weights with the squared multi- 
ple correlation coefficient serving as an estimate of the accuracy of the model 
linking the independent variables to the criterion. Points three and four amount 
to a suggestion to use two of four research strategies proposed by Lina & Weit 
(1969). Researchers also need to make efforts to determine the maximum amount 
of variance which could be explained by differences between schools or classrooms. 
They could then est ima te the effectiveness of their variables in representing 
those differences. Above all, researchers need to be careful to point out which 
sort of variance their statistical models explain. All too often, models explain 
variance or represent a theory which is different than that which the researcher 



wishes to study. 
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